Effective Processing of Metadata Operations in Large- Scale Distributed File Systems
نویسندگان
چکیده
When metadata is distributed across many metadata servers in a distributed file system with a locality-preserving metadata distribution unit like a directory, simply increasing the number of metadata servers does not give high-aggregate performance for the metadata operation such as file creation. This paper proposes an effective and consistent method for processing metadata operations, at low cost, without depending on the traditional 2 phase commit protocol. Designing an alternate protocol, we showed that the performance limitations of metadata operations, caused by the distributed environment, can be overcome and the materialization of a large-scale distributed file system is actually possible.
منابع مشابه
MetaFlow: a Scalable Metadata Lookup Service for Distributed File Systems in Data Centers
In large-scale distributed file systems, efficient metadata operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its significant CPU overhead. Our investigations showed that the lookup service could reduce system thro...
متن کاملScalable Storage for Data-Intensive Computing
Cloud computing applications require a scalable, elastic and fault tolerant storage system. We survey how storage systems have evolved from the traditional distributed filesystems, peer-to-peer storage systems and how these ideas have been synthesized in current cloud computing storage systems. Then, we describe how metadata management can be improved for a file system built to support large sc...
متن کاملScalable Archival Data and Metadata Management in Object-based File Systems
Online archival capabilities like snapshots or checkpoints are fast becoming an essential component of robust storage systems. Emerging large distributed file systems are also shifting to object-based storage architectures that decouple metadata from file I/O operations. As the size of such systems scale to petabytes of storage, it is critically important that file system features continue to o...
متن کاملCalvinFS: Consistent WAN Replication and Scalable Metadata Management for Distributed File Systems
Existing file systems, even the most scalable systems that store hundreds of petabytes (or more) of data across thousands of machines, store file metadata on a single server or via a shared-disk architecture in order to ensure consistency and validity of the metadata. This paper describes a completely different approach for the design of replicated, scalable file systems, which leverages a high...
متن کاملA Metadata-Rich File System
Despite continual improvements in the performance and reliability of large scale file systems, the management of file system metadata has changed little in the past decade. The mismatch between the size and complexity of large scale data stores and their ability to organize and query their metadata has led to a de facto standard in which raw data is stored in traditional file systems, while rel...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2016